Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dictionary Benchmarks #2705

Closed
wants to merge 18 commits into from
Closed

Dictionary Benchmarks #2705

wants to merge 18 commits into from

Conversation

mregen
Copy link
Contributor

@mregen mregen commented Aug 10, 2024

Proposed changes

  • benchmarks for use cases to read from dictionaries with key uint, NodeId and ExpandedNodeId
  • validate speed of NodeIdDictionary compared to native Dictionary<NodeId,value>

Related Issues

  • Fixes #

Types of changes

What types of changes does your code introduce?
Put an x in the boxes that apply. You can also fill these out after creating the PR.

  • Bugfix (non-breaking change which fixes an issue)
  • Enhancement (non-breaking change which adds functionality)
  • Test enhancement (non-breaking change to increase test coverage)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected, requires version increase of Nuget packages)
  • Documentation Update (if none of the other choices apply)

Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

  • I have read the CONTRIBUTING doc.
  • I have signed the CLA.
  • I ran tests locally with my changes, all passed.
  • I fixed all failing tests in the CI pipelines.
  • I fixed all introduced issues with CodeQL and LGTM.
  • I have added tests that prove my fix is effective or that my feature works and increased code coverage.
  • I have added necessary documentation (if appropriate).
  • Any dependent changes have been merged and published in downstream modules.

Further comments

NodeId Dictionary Benchmarks: For each IdType, last one is with mixed NodeId.

Method Runtime NodeIdType Mean Error StdDev Median Ratio RatioSD Gen0 Allocated Alloc Ratio
TestConcurrentDictionary .NET 6.0 Numeric 46.02 us 0.870 us 2.118 us 45.59 us 0.36 0.02 0.2441 2496 B 0.09
TestFrozenDictionary .NET 8.0 Numeric 53.18 us 1.054 us 1.172 us 53.29 us 0.41 0.01 0.2441 2496 B 0.09
TestConcurrentDictionary .NET 8.0 Numeric 60.14 us 1.198 us 1.864 us 60.17 us 0.46 0.01 0.2441 2496 B 0.09
TestRegularDictionary .NET 8.0 Numeric 62.43 us 1.234 us 1.605 us 62.12 us 0.48 0.02 0.2441 2496 B 0.09
TestReadonlyDictionary .NET 8.0 Numeric 70.86 us 1.369 us 1.521 us 70.60 us 0.55 0.02 0.2441 2496 B 0.09
TestRegularDictionary .NET 6.0 Numeric 76.49 us 1.427 us 1.335 us 76.70 us 0.59 0.02 0.2441 2496 B 0.09
TestReadonlyDictionary .NET 6.0 Numeric 78.02 us 1.414 us 1.513 us 77.70 us 0.60 0.02 0.2441 2496 B 0.09
TestFrozenDictionary .NET 6.0 Numeric 78.73 us 1.544 us 3.153 us 78.23 us 0.62 0.02 0.2441 2496 B 0.09
TestImmutableDictionary .NET 6.0 Numeric 110.08 us 2.133 us 2.095 us 110.05 us 0.85 0.03 0.2441 2496 B 0.09
TestFrozenDictionary .NET Framework 4.8 Numeric 120.70 us 2.322 us 3.179 us 119.90 us 0.93 0.03 4.1504 27152 B 1.00
TestReadonlyDictionary .NET Framework 4.8 Numeric 124.86 us 2.485 us 4.905 us 123.08 us 0.98 0.04 4.2725 27152 B 1.00
TestConcurrentDictionary .NET Framework 4.8 Numeric 126.45 us 2.516 us 3.359 us 126.27 us 0.98 0.04 4.2725 27441 B 1.01
TestRegularDictionary .NET Framework 4.8 Numeric 129.84 us 2.471 us 2.845 us 129.97 us 1.00 0.00 4.2725 27152 B 1.00
TestImmutableDictionary .NET 8.0 Numeric 157.73 us 2.583 us 2.289 us 156.87 us 1.22 0.03 0.2441 2496 B 0.09
TestNodeIdDictionary .NET 6.0 Numeric 163.12 us 3.025 us 7.363 us 161.98 us 1.26 0.05 - - 0.00
TestNodeIdDictionary .NET 8.0 Numeric 176.36 us 3.513 us 3.608 us 177.84 us 1.36 0.04 - - 0.00
TestImmutableDictionary .NET Framework 4.8 Numeric 238.51 us 4.722 us 5.437 us 237.57 us 1.84 0.07 4.1504 27152 B 1.00
TestNodeIdDictionary .NET Framework 4.8 Numeric 243.84 us 4.697 us 4.394 us 245.12 us 1.88 0.05 - - 0.00
TestSortedDictionary .NET 8.0 Numeric 375.62 us 6.593 us 6.167 us 374.91 us 2.90 0.08 2.4414 20928 B 0.77
TestSortedDictionary .NET 6.0 Numeric 418.77 us 14.805 us 43.420 us 432.05 us 2.76 0.36 2.4414 20928 B 0.77
TestSortedDictionary .NET Framework 4.8 Numeric 613.54 us 11.245 us 12.032 us 613.00 us 4.74 0.18 2.9297 20992 B 0.77
TestFrozenDictionary .NET 8.0 String 135.02 us 3.761 us 11.090 us 135.64 us 0.77 0.04 - - 0.000
TestConcurrentDictionary .NET 8.0 String 143.73 us 2.831 us 6.728 us 141.59 us 0.76 0.03 - - 0.000
TestConcurrentDictionary .NET 6.0 String 146.10 us 2.842 us 4.254 us 145.47 us 0.79 0.02 - - 0.000
TestFrozenDictionary .NET 6.0 String 156.07 us 2.980 us 3.061 us 156.01 us 0.84 0.03 - - 0.000
TestReadonlyDictionary .NET 6.0 String 162.67 us 3.252 us 6.643 us 161.44 us 0.87 0.05 - - 0.000
TestRegularDictionary .NET 6.0 String 163.73 us 3.265 us 8.071 us 165.55 us 0.84 0.03 - - 0.000
TestRegularDictionary .NET 8.0 String 164.39 us 3.102 us 3.319 us 164.84 us 0.88 0.03 - - 0.000
TestReadonlyDictionary .NET 8.0 String 168.96 us 3.206 us 2.842 us 168.84 us 0.91 0.03 - - 0.000
TestRegularDictionary .NET Framework 4.8 String 186.34 us 3.607 us 3.543 us 186.36 us 1.00 0.00 3.9063 24650 B 1.000
TestConcurrentDictionary .NET Framework 4.8 String 197.70 us 3.952 us 7.893 us 198.54 us 1.06 0.03 3.9063 24650 B 1.000
TestFrozenDictionary .NET Framework 4.8 String 202.71 us 4.361 us 12.790 us 203.22 us 1.02 0.05 3.9063 24650 B 1.000
TestReadonlyDictionary .NET Framework 4.8 String 203.31 us 5.884 us 17.165 us 207.10 us 1.01 0.07 3.9063 24650 B 1.000
TestImmutableDictionary .NET 8.0 String 227.18 us 3.241 us 3.032 us 226.47 us 1.22 0.03 - - 0.000
TestImmutableDictionary .NET 6.0 String 242.83 us 4.811 us 11.528 us 237.98 us 1.33 0.07 - - 0.000
TestImmutableDictionary .NET Framework 4.8 String 332.02 us 6.621 us 11.596 us 332.92 us 1.80 0.06 3.9063 24652 B 1.000
TestSortedDictionary .NET 8.0 String 358.20 us 6.852 us 8.666 us 357.63 us 1.92 0.05 - - 0.000
TestSortedDictionary .NET 6.0 String 435.37 us 8.357 us 11.156 us 434.54 us 2.35 0.06 - - 0.000
TestSortedDictionary .NET Framework 4.8 String 614.62 us 17.939 us 52.328 us 590.99 us 3.62 0.19 - - 0.000
TestNodeIdDictionary .NET 8.0 String 823.99 us 16.037 us 22.481 us 822.40 us 4.48 0.10 - 1 B 0.000
TestNodeIdDictionary .NET 6.0 String 894.55 us 17.473 us 31.950 us 889.88 us 4.87 0.22 - 1 B 0.000
TestNodeIdDictionary .NET Framework 4.8 String 1,615.54 us 31.901 us 65.882 us 1,600.85 us 8.78 0.45 - - 0.000
TestFrozenDictionary .NET 8.0 Guid 54.10 us 1.071 us 1.985 us 53.48 us 0.42 0.02 0.3662 3072 B 0.11
TestConcurrentDictionary .NET 8.0 Guid 58.43 us 1.164 us 2.429 us 57.93 us 0.46 0.03 0.3662 3072 B 0.11
TestRegularDictionary .NET 8.0 Guid 62.49 us 1.184 us 1.050 us 62.59 us 0.49 0.02 0.3662 3072 B 0.11
TestConcurrentDictionary .NET 6.0 Guid 66.31 us 1.307 us 2.111 us 66.10 us 0.51 0.03 0.3662 3072 B 0.11
TestReadonlyDictionary .NET 8.0 Guid 76.34 us 1.812 us 5.110 us 76.19 us 0.62 0.05 0.3662 3072 B 0.11
TestFrozenDictionary .NET 6.0 Guid 81.33 us 1.587 us 1.890 us 81.72 us 0.63 0.02 0.3662 3072 B 0.11
TestRegularDictionary .NET 6.0 Guid 81.67 us 1.023 us 0.907 us 81.70 us 0.64 0.03 0.3662 3072 B 0.11
TestReadonlyDictionary .NET 6.0 Guid 90.40 us 1.794 us 3.095 us 90.91 us 0.70 0.03 0.3662 3072 B 0.11
TestConcurrentDictionary .NET Framework 4.8 Guid 124.41 us 2.470 us 4.989 us 123.78 us 0.97 0.06 4.3945 28116 B 1.01
TestRegularDictionary .NET Framework 4.8 Guid 128.71 us 2.509 us 4.650 us 128.39 us 1.00 0.00 4.3945 27730 B 1.00
TestReadonlyDictionary .NET Framework 4.8 Guid 130.34 us 2.556 us 5.164 us 129.36 us 1.01 0.05 4.3945 27730 B 1.00
TestFrozenDictionary .NET Framework 4.8 Guid 134.47 us 2.595 us 2.427 us 133.59 us 1.05 0.06 4.3945 27730 B 1.00
TestImmutableDictionary .NET 8.0 Guid 153.56 us 3.066 us 5.370 us 152.48 us 1.19 0.07 0.2441 3072 B 0.11
TestImmutableDictionary .NET 6.0 Guid 161.10 us 2.810 us 2.760 us 161.45 us 1.26 0.06 0.2441 3072 B 0.11
TestNodeIdDictionary .NET 8.0 Guid 183.89 us 3.645 us 8.448 us 181.92 us 1.43 0.09 - - 0.00
TestImmutableDictionary .NET Framework 4.8 Guid 244.24 us 4.608 us 8.310 us 243.36 us 1.90 0.09 4.3945 27732 B 1.00
TestNodeIdDictionary .NET 6.0 Guid 253.39 us 3.120 us 4.475 us 253.11 us 1.97 0.09 - - 0.00
TestNodeIdDictionary .NET Framework 4.8 Guid 271.99 us 5.007 us 4.684 us 272.77 us 2.12 0.12 - - 0.00
TestSortedDictionary .NET 8.0 Guid 368.07 us 7.347 us 15.173 us 363.62 us 2.86 0.17 2.9297 27488 B 0.99
TestSortedDictionary .NET 6.0 Guid 477.58 us 9.524 us 22.635 us 472.65 us 3.79 0.20 2.9297 25504 B 0.92
TestSortedDictionary .NET Framework 4.8 Guid 650.33 us 12.952 us 25.262 us 647.95 us 5.07 0.31 3.9063 24816 B 0.89
TestConcurrentDictionary .NET 8.0 Opaque 102.93 us 2.073 us 6.048 us 101.21 us 0.28 0.02 - - 0.00
TestFrozenDictionary .NET 8.0 Opaque 106.72 us 2.116 us 2.896 us 106.08 us 0.28 0.01 - - 0.00
TestRegularDictionary .NET 8.0 Opaque 122.29 us 3.391 us 9.839 us 121.31 us 0.30 0.01 - - 0.00
TestReadonlyDictionary .NET 8.0 Opaque 126.72 us 3.258 us 9.605 us 127.71 us 0.31 0.03 - - 0.00
TestConcurrentDictionary .NET 6.0 Opaque 148.60 us 3.363 us 9.864 us 146.01 us 0.42 0.02 - - 0.00
TestFrozenDictionary .NET 6.0 Opaque 162.26 us 4.935 us 14.474 us 160.73 us 0.47 0.03 - - 0.00
TestRegularDictionary .NET 6.0 Opaque 166.95 us 3.302 us 5.141 us 165.73 us 0.44 0.02 - - 0.00
TestReadonlyDictionary .NET 6.0 Opaque 167.89 us 4.883 us 14.399 us 172.22 us 0.43 0.05 - - 0.00
TestImmutableDictionary .NET 8.0 Opaque 176.84 us 3.302 us 3.089 us 177.43 us 0.47 0.02 - - 0.00
TestNodeIdDictionary .NET 8.0 Opaque 234.29 us 4.644 us 13.472 us 231.34 us 0.64 0.04 - - 0.00
TestImmutableDictionary .NET 6.0 Opaque 244.60 us 7.963 us 23.479 us 247.82 us 0.69 0.04 - - 0.00
TestNodeIdDictionary .NET Framework 4.8 Opaque 304.79 us 5.988 us 5.881 us 304.74 us 0.81 0.03 - - 0.00
TestNodeIdDictionary .NET 6.0 Opaque 305.39 us 5.834 us 7.586 us 306.31 us 0.81 0.03 - - 0.00
TestSortedDictionary .NET 8.0 Opaque 338.66 us 7.589 us 22.017 us 340.17 us 0.93 0.04 - - 0.00
TestConcurrentDictionary .NET Framework 4.8 Opaque 377.72 us 7.506 us 19.106 us 370.30 us 1.00 0.07 3.4180 24576 B 1.00
TestRegularDictionary .NET Framework 4.8 Opaque 378.33 us 7.545 us 9.265 us 382.16 us 1.00 0.00 3.4180 24576 B 1.00
TestSortedDictionary .NET 6.0 Opaque 391.46 us 7.805 us 16.291 us 389.88 us 1.03 0.06 - - 0.00
TestReadonlyDictionary .NET Framework 4.8 Opaque 391.50 us 7.281 us 18.665 us 386.56 us 1.05 0.07 3.4180 24576 B 1.00
TestFrozenDictionary .NET Framework 4.8 Opaque 403.28 us 10.045 us 29.617 us 392.75 us 1.02 0.05 3.4180 24576 B 1.00
TestImmutableDictionary .NET Framework 4.8 Opaque 577.36 us 11.368 us 17.361 us 578.98 us 1.53 0.06 3.4180 24576 B 1.00
TestSortedDictionary .NET Framework 4.8 Opaque 606.99 us 15.433 us 44.774 us 593.81 us 1.70 0.15 - - 0.00
TestFrozenDictionary .NET 8.0 4 95.13 us 1.738 us 1.541 us 95.16 us 0.67 0.03 0.1221 1816 B 0.07
TestRegularDictionary .NET 8.0 4 105.59 us 1.799 us 2.522 us 105.92 us 0.75 0.03 0.1221 1816 B 0.07
TestConcurrentDictionary .NET 8.0 4 106.78 us 2.064 us 2.684 us 106.58 us 0.75 0.03 0.1221 1816 B 0.07
TestConcurrentDictionary .NET 6.0 4 115.67 us 2.118 us 2.828 us 115.32 us 0.82 0.03 0.1221 1816 B 0.07
TestReadonlyDictionary .NET 8.0 4 116.61 us 2.188 us 3.655 us 116.57 us 0.83 0.04 - 1816 B 0.07
TestFrozenDictionary .NET 6.0 4 122.56 us 2.271 us 2.124 us 123.13 us 0.86 0.03 - 1816 B 0.07
TestRegularDictionary .NET 6.0 4 124.23 us 1.942 us 1.817 us 124.25 us 0.87 0.03 - 1816 B 0.07
TestReadonlyDictionary .NET 6.0 4 128.39 us 1.792 us 1.589 us 128.25 us 0.90 0.03 - 1816 B 0.07
TestFrozenDictionary .NET Framework 4.8 4 136.65 us 6.277 us 18.409 us 140.58 us 1.07 0.08 4.1504 26470 B 1.00
TestRegularDictionary .NET Framework 4.8 4 141.05 us 2.714 us 4.144 us 141.56 us 1.00 0.00 4.1504 26470 B 1.00
TestConcurrentDictionary .NET Framework 4.8 4 143.54 us 2.785 us 4.082 us 143.05 us 1.02 0.04 4.1504 26792 B 1.01
TestReadonlyDictionary .NET Framework 4.8 4 177.78 us 3.260 us 3.624 us 177.66 us 1.25 0.04 4.1504 26470 B 1.00
TestImmutableDictionary .NET 6.0 4 200.83 us 5.146 us 14.597 us 198.97 us 1.42 0.13 - 1816 B 0.07
TestImmutableDictionary .NET 8.0 4 209.96 us 4.140 us 4.066 us 209.23 us 1.48 0.06 - 1816 B 0.07
TestImmutableDictionary .NET Framework 4.8 4 273.35 us 7.674 us 22.626 us 266.27 us 2.04 0.18 3.9063 26472 B 1.00
TestNodeIdDictionary .NET 6.0 4 360.79 us 11.721 us 34.375 us 354.38 us 2.60 0.23 - - 0.00
TestNodeIdDictionary .NET 8.0 4 364.32 us 6.955 us 6.506 us 363.54 us 2.56 0.09 - - 0.00
TestSortedDictionary .NET 8.0 4 411.75 us 7.028 us 8.888 us 412.44 us 2.91 0.10 1.4648 14184 B 0.54
TestSortedDictionary .NET 6.0 4 505.42 us 9.949 us 12.936 us 503.34 us 3.57 0.11 0.9766 14249 B 0.54
TestNodeIdDictionary .NET Framework 4.8 4 590.35 us 11.620 us 20.654 us 588.00 us 4.22 0.18 - - 0.00
TestSortedDictionary .NET Framework 4.8 4 602.88 us 14.142 us 40.803 us 592.73 us 4.18 0.29 1.9531 14200 B 0.54

Copy link

codecov bot commented Aug 10, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 55.33%. Comparing base (f001eab) to head (90a9c2d).
Report is 3 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2705      +/-   ##
==========================================
+ Coverage   55.30%   55.33%   +0.02%     
==========================================
  Files         352      352              
  Lines       67326    67326              
  Branches    13806    13806              
==========================================
+ Hits        37235    37253      +18     
+ Misses      26005    25993      -12     
+ Partials     4086     4080       -6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.


/// <summary>
/// Tests performance and memory usage of ImmutableDictionary.
/// </summary>
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

comment

public abstract class DictionaryKeyBenchmark<T>
{
// [Params(16, 128, 1024)]
public int NumElements { get; set; } = 1024;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re add size parameter

public int NumElements { get; set; } = 1024;

// Create a Dictionary and a SortedDictionary
protected T[] m_lookup = null;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

comment


/// <summary>
/// Tests performance and memory usage of SortedDictionary.
/// </summary>
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

comment

mregen and others added 5 commits August 12, 2024 10:34
@CLAassistant
Copy link

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ mregen
❌ dependabot[bot]
You have signed the CLA already but the status is still pending? Let us recheck it.

@mregen mregen closed this Oct 17, 2024
@mregen mregen deleted the dictbenchmarks branch October 17, 2024 17:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants